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EXECUTIVE SUMMARY 



The general purpose of this study is to assess the effectiveness of Milwaukee’s independent charter schools in 
promoting student achievement growth. Independent charter schools are authorized hy non-school district 
entities and are considered “independent” because they are not a part of the Milwaukee Public School District 
(MPS). Throughout the course of this report we will estimate three-year achievement growth for independent 
charter school students who were in grades 3 through 8 at baseline (2006-07). We will examime four years of 
scores in reading and math on the Wisconsin Knowledge and Concepts Examination (WKCE). Specifically, the 
report presents the results of an analysis comparing achievement gains of independent charter students to the 
achievement gains of a group of matched comparison students attending MPS. Our next report, to be released 
in spring 2012, will examine four-year achievement gains. 

This report draws upon a panel of 2,295 students attending 10 of Milwaukee’s 14 independent charter schools 
in grades 3-8 in 2006-07 with test scores for that year. The four charter schools excluded from the sample either 
were not open for both the baseline and outcome years or did not enroll students in tested grades. The 2,295 
tested Milwaukee independent charter school students were carefully matched to an identical-sized sample of 
students attending MPS to provide a comparison group against which the achievement gains of independent 
charter students could be assessed. Students were matched on prior achievement and propensity scores, which 
help to control for differences between students on observable characteristics. We are confident our matching 
algorithm produced a charter and MPS sample equivalent on prior achievement. However, similar to other 
observational studies, our study is unable to control for all potential sources of unobservable selection bias. We 
believe this potential threat is less of a concern because highly motivated parents in MPS, similar to parents of 
charter school students, have many alternative options to exercise choice within the Milwaukee Public School 
system. 

The basic conclusions of this report are: 

• Based on three years of student achievement growth, charter school students outperformed MPS 
students in both reading and mathematics after controlling for baseline achievement and other student 
characteristics. These results were statistically significant with more than 99 percent confidence. 

• Drawing upon our earlier study of charter school gains (Witte et ah, 2010), we see a clear pattern of 
positive charter school effects growing over time. There was little consistent evidence of differences 
in achievement gains between charter and MPS students after one year. The second year growth was 
better for charters in some models and for some tests, but not for others. After three years, a sizable 
independent charter school advantage was apparent in all of our analyses. 
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• Supplementary analysis revealed that conversion independent charter schools - schools that converted 
from private schools^ - as a subgroup did better than MPS schools in all three growth years. In the third 
growth year non-conversion charter schools also did better than MPS schools in mathematics but not 
necessarily in reading. 

• Further analyses indicate that after three years charter schools appear to have the greatest positive impact 
on students at the lower end of the achievement distribution. 

• When looking specifically at “stayer students”, or those who remained in their initial sector (charter or 
traditional) for four years, the performance of charter stayers was much higher than that of MPS stayers 
after three years. 

We are appreciative of the constructive comments on a preliminary draft that we received from outside experts 
as well as the School Choice Demonstration Project Research Advisory Board and research team. Additionally, 
we thank Russell Diamond for advice on data coding. All remaining errors are the responsibility of the 
authors alone. 

This project is being funded by a diverse set of philanthropies including the Annie E. Casey, Joyce, Kern Family, 
Lynde and Harry Bradley, Robertson, and Walton Family foundations. We thank them for their generous 
support and acknowledge that the actual content of this report is solely the responsibility of the authors 
and does not necessarily refiect any official positions of the various funding organizations, the University of 
Arkansas, or the University of Wisconsin. We also express our gratitude to officials at MPS, the independent 
charter schools, and the State Department of Public Instruction for their willing cooperation, advice, 
and assistance. 



1 Most studies discussing conversion charter schools define them as traditional public schools which converted to charter status 
(Loveless etal., 2006; Sass, 2006). Peterson etal. (2006) expands on this definition to include private schools which converted 
to charter status as another form of conversion charter school. We adopt this definition for our study. In addition, the paper on 
school choice in Milwaukee by Peterson etal. (2006) confirms that the 10 charter schools in our study either converted from a 
private school or began as a new school. 
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Charter Schools 

Charter schools are tuition-free public schools that are authorized to operate within an agreed “charter.” The 
charters often specify the size of the school, its mission, specialized curricula and pedagogy, unique personnel 
practices, and specific goals that the school must meet over time in order to he reauthorized. Most charter 
schools use an open enrollment system that permits students to attend the school even if they do not live close 
hy. Thus, charter schools are subject to parental school choice. To facilitate these unique schools, they are often 
given waivers from some of the administrative and accountability requirements of other public schools. This 
does not exempt charter schools from the testing and reporting requirements of the federal No Child Left 
Behind law. 

Since the early 1990s, when the first charter school opened, the number of these schools has increased 
dramatically. We expect to see even further expansion of charter schools in the coming years as a result of the 
recent “Race to the Top” initiative, which required states to relax charter school laws to be competitive for federal 
education funds. 

Similar to national trends, the number of charter schools in Wisconsin has grown widely, from 17 in 1997 
to 206 in 2009 (Evers et al, 2009). Charter schools serve more than 37,000 students in the state (Center 
for Education Reform, 2009; Evers et al, 2009). Government officials see the potential of charter schools as 
part of a reform to transform public education in the state. For the first time. Governor Jim Doyle and State 
Superintendent Tony Evers attended the Wisconsin Charter Schools Conference in April, 2009 (Borsuk, 2009). 
In October of 2009, President Barack Obama and Education Secretary Arne Duncan visited Wright Middle 
School, a charter school in Madison, to highlight the role of charter schools in the “Race to the Top” initiative. 
Wisconsin has also received S86 million in federal funding over the next five years to support charter schools in 
Milwaukee and the state by allocating grants to new and existing charter schools. 

This longitudinal study evaluates the impact of independent charter schools on student achievement in 
Milwaukee, Wisconsin over four years. Milwaukee is one of the few places in the U.S. that contains both 
district-authorized charter schools and independent charter schools (Table 1). As of the beginning of this study, 
38 district-authorized charter schools remain part of the Milwaukee Public School system. Of these 38 district- 
authorized charters, a total of 25 are staffed by teachers who remain employees of the school district and bound 
by the union-negotiated collective bargaining agreement. These schools are referred to as “instrumentality” 
charters. The remaining 13 MPS “non-instrumentality” charter schools are permitted to hire and employ 
nonunion teachers. 
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Table 1 . Types of Public Charter Schools in Milwaukee, Wl, 2006-2007 



Type 


Number 


Percentage of All 


MPS Instrumentality 


25 


48.1 


MPS Non-Instrumentality 


13 


25.0 


MPS Total 


38 


73.1 


Independent U ofW-Milwaukee 


9 


17.3 


Independent City of Milwaukee 


5 


9.6 


Independent Total 


14 


26.9 



Source: Wisconsin Charter Schoois Yearbook 2006-2007 
http://dpi.state.wi.us/sms/pdf/2006-07yearbook.pcif 



In Milwaukee, charter schools are one among a wide variety of school choice options including charter and magnet 
schools affiliated with MPS, open enrollment into other public school districts, and private schools accepting 
vouchers under the Milwaukee Parental Choice Program. In 2006-07, charter schools in Milwaukee comprised close 
to a quarter of the charter schools in the state. 

As discussed previously, independent charters are a distinctive type of charter school in Milwaukee. They were 
created hy 1997 legislation to he authorized hy the City of Milwaukee Common Council, the University of 
Wisconsin-Milwaukee (UWM), the Milwaukee Area Technical College, or the University of Wisconsin at Parkside 
(Racine). They are not connected to MPS. Of the 9 UWM and 5 City of Milwaukee independent charters open in 
2006-07, 10 are the subjects of this research. The student enrollments by grade for the baseline year of 2006-07 for 
our school sample are indicated in Table 2. As is apparent, UWM charter schools have many more students than 
City charters, and there are very few students in grade 9 compared to grades 3 to 8.^ 



Table 2. Milwaukee Independent Charter School Sample Enrollment, 2006-07 



Grade 


Schools 


3 


4 


5 


6 


7 


8 


9 


TOTAL 


UWM 


6 


328 


331 


338 


287 


241 


239 


140 


1904 


City 


4 


92 


89 


99 


88 


119 


80 


58 


625 


TOTAL 


10 


420 


420 


437 


375 


360 


319 


198 


2529 



Source: Charter Schools page on the Department of Public Instruction website: http://www.dpi.state.wi.us/sms/xls/0607enrl.htm 

Note: For the 2006-07 baseline year, there are no test score data for structural reasons for the following schools: Inlands Sea School of 
Expeditionary Learning (ISSEL), Milwaukee Renaissance Academy (MRA), Seeds of Health (SoHE), School for Early Development (SEDA), 
and Massai Institute which has closed as of 2007-08. For the first four schools, they did not test in November 2006 when schools typically 
test because they were not yet open. ISSEL opened in 01/2006, MRA in 08/2007, and SoHE in 08/2007. SEDA is an early education school 
with grade levels K4-2 and does not have data for grades 3-8, or grade 10 because it does not have these grade levels at its school. Bruce 
Guadalupe Community School transitioned from the oversight of MPS to being authorized as a charter by UWM in 2009-2010. In addition, in 
the current 2010-2011 school year, the City chartered King's Academy while UWM chartered Urban Day School, Veritas High School and a 
new campus of Milwaukee College Preparatory School, Lindsay Heights. Many of the students attending Lindsay Heights formerly attended 
the Academy of Learning and Leadership which closed in September 2010. 



2 Ninth grade students are not included in this study because tests are not given past the 10th grade. 
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Research on Charter Schools 

Supporters see the potential of high-quality charter schools to help transform the education system hy raising 
achievement levels, closing achievement gaps, placing competitive pressure on traditional public schools and 
stimulating greater innovation. They posit that giving charter schools more flexibility over such practices as 
hiring teachers, budgeting school funds, and selecting curricula will lead to these positive outcomes (Finn, 
Manno, & Vanourek, 2001; Payne & Knowles, 2009). Further, through a system of accountability, they expect 
to reduce the number of low-quality charter schools that are not able to meet the standards they agreed to in 
their charters. 

In contrast, critics are concerned about charter schools drawing away resources from traditional public schools 
(e.g. teachers, funding, and motivated students), increasing racial segregation, and lacking the accountability 
structure to close or improve low-quality charter schools (Wells et al, 2002). They fear charters are performing 
no better and sometimes worse than traditional public schools. To date the research on the performance of 
charter schools is mixed, ranging from negative, neutral, mildly positive, to a few specific studies which are 
strongly positive (Bifulco & Ladd, 2006; Sass, 2006; Ballou et al, 2006; Hanushek et al, 2007; Booker et 
al.,2007', Zimmer et al., 2009; Witte et al, 2007; Witte & Lavertu, 2009; CREDO, 2009; Hoxby et a/., 2009; 
CREDO, 2010; Abdulkadiroglu et al, 2009; Tuttle a/., 2010; Gleason et al, 2010). 

Most of these prior studies are observational, with a smaller number of studies employing randomized 
experimental designs. While randomized experiments are recognized as producing gold standard results, they 
are costly and logistically less feasible than securing existing longitudinal data used in observational studies. In 
general, where there have been positive or negative impacts of charter schools, they have typically been small 
in magnitude (Hill et al, 2006). The exceptions to this general trend are some more recent randomized trials of 
charter schools in Boston and New York (Abdulkadiroglu et al, 2009; Hoxby et al, 2009) and an observational 
study of those charters affiliated with the Knowledge is Power Program (KIPP) (Tuttle et al, 2010) which And 
strong positive charter impacts. Research on charter impacts in Milwaukee more closely resembles the general 
trend of charter school performance, with two prior observational studies showing modestly positive gains in 
math for charter schools authorized by Milwaukee Public Schools (Witte et al, 2007; Witte & Lavertu, 2009). 
Similarly, our report of one-year student achievement growth of students in Milwaukee independent charter 
schools found little difference in achievement between these schools and MPS schools (Witte et al, 2010). 
However, the report highlighted that students in conversion charter schools - i.e. formerly private schools - did 
have more positive achievement growth than students in traditional MPS schools. 

In this report, we extend the initial study and evaluate two- and three-year achievement gains. Using four years 
of panel data — 2006-07 to 2009-10 — ^we estimate models of achievement gains for charter school students who 
were in grades 3 through 8 at baseline (2006-07), relative to similar students in MPS, controlling for baseline 
test scores and student characteristics. 
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Research Question 

Through this evaluation we endeavor to understand whether students benefit in the short term and the long 
term from attending an independent charter school. In this report the primary research question is: Do 
Milwaukee’s independent charter schools produce higher rates of two- and three-year achievement gains than 
do Milwaukee public schools? For purposes of this study, achievement is measured by performance on the 
reading and mathematics sections of the Wisconsin Knowledge and Concepts Examination (WKCE) that 
all public school students are required to take in grades 3 through 8 and 10. The WKCE is administered in 
the fall of each school year and uses short answer and multiple choice questions to test student mastery in 
reading, math, language arts, science and social studies. Scores on these examinations are recorded in both 
scale (or developmental) scores and proficiency levels. We rely on scale scores in this analysis. As indicated 
below, we standardize these scale scores in order to allow comparisons across grade levels. Student test scores 
and demographic characteristics used in this study were provided by the Office of Research and Evaluation at 
MPS, the Office of Charter Schools at the University ofWisconsin-Milwaukee, and the City of Milwaukee 
Common Council.^ 

Matched Samples 

The first step in our analysis involved determining the comparative samples of students. Because the total 
number of students in independent charters for which test scores were available in 2006-07 was 2,295, we 
decided to include all of those students in the charter school sample. The issue was then how to create a relevant 
matched sample that would be similar on important observed characteristics at baseline. To do that we first 
selected a random sample of MPS students matched by grade. In doing so we discovered that the baseline test 
scores (November 2006) for that group differed from those in the independent charter schools in a number of 
grades. The random MPS sample of students usually scored higher than the independent charter students. Thus 
the random sample would have started out students at different levels of prior achievement. 

To adjust for this problem we undertook a two-step procedure.'* First, each student in the charter sample was 
matched with the set of MPS students in their grade with baseline WKCE test scores within five percent 



3 We are particularly grateful to Deb Lindsey of MPS, Robert Kattman of UW-M, and Cindy Zautcke of the Common Council for 
their support and assistance in obtaining the necessary data. All student data were provided to us absent personal information 
about the student, such as name and address, or such "personal signifiers"were deleted from the data prior to analysis. 

4 See Witte eta!., 2010 for figures and tables describing the results of our matching protocol. 
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of their score. This was done within 20 hands from the lowest to the highest based on the distribution of 
independent charter student combined reading and math test scores. Second, the charter panelist and each 
MPS student within that five percent “grade band” were assigned a “propensity score” that predicted their 
likelihood of being in a charter school based on race, gender, English Language Learner (ELL) status, and 
participation in the federal Free/Reduced Price Lunch (FRL) program. The MPS student within the grade 
band with the charter school propensity score closest to the propensity score of a given student in the charter 
panel was drawn out of the panel (without replacement) and became a member of the MPS comparison sample. 
The result of all these matches was a sample of 2,295 independent charter school students and 2,295 MPS 
comparison students that closely resemble the charter school students on baseline test scores and other factors 
that predict charter school enrollment. The purpose of this procedure was to reduce the differences in observed 
characteristics between the independent charter students and a random sample of MPS students. 

There are few statistically significant mean differences in baseline reading and math scale scores when 
comparing the Independent Charter Sample and the MPS Matched Sample. Only in 4* and 6* grade math are 
there any baseline test score differences. Both of those differences between the charter and matched samples 
are statistically significant only at the 90 percent confidence level, the lowest confidence level that we use in this 
evaluation. This suggests the matching was successful. Thus, in terms of prior achievement we have created the 
proverbial apples-to-apples comparison to begin our study (Witte et al, 2010). 

Our matching algorithm also produced charter and MPS student samples that are generally similar regarding 
other important measurable student characteristics. The two samples are very close on race and gender 
demographics. The MPS Matched sample does differ significantly from the Independent Charter Sample 
regarding populations of exceptional education^ and free lunch students, though a random sample of MPS 
students would have differed from the charter sample even more regarding these two student characteristics (See 
Witte et «/., 2010). Because of these differences, in most of the analyses to follow, we independently control for 
all of these student characteristics in our most precise regression models.*^ 



5 We ran a sensitivity analysis predicting the likelihood of a student having an exceptional education status. The fixed effects 
regression results showed that charter coefficient was not significant meaning that students in both sectors had an equal 
likelihood of being classified as having a disability. 

6 The initial difference between the charter and matched sample on free lunch status is due to incomplete free lunch data counts 
in a few schools. We correct for this in our models in two ways. If a student had a free lunch observation in 2007-08, 2008-09 

or 2009-10, we backfilled the data. In addition, for students with missing data on free-lunch or any other control variable, we 
include an indicator in our models controlling for this missing data. Doing so allows our regression models to draw upon the 
actual data in each student observation, and only that actual data, to inform the coefficient estimates of the model (Cohen & 
Cohen, 1983). 
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The matching design and baseline control variables limit the extent to which measurable student characteristics 
might bias our analysis of independent charter and MPS student test score gains. Because students were not 
randomly assigned to the two groups, however, we cannot rule out unmeasured student characteristics as a 
potential source of bias. For example, if the students in Milwaukee independent charter schools are similar to 
our matched MPS sample in most ways except that they have more motivated parents, as demonstrated by the 
fact that they enrolled the student in a school of choice, then the charter students might demonstrate stronger 
achievement gains simply due to such a “self-selection” bias. On the other hand, if parents seek alternatives to 
their neighborhood public school primarily when their child is struggling or exhibiting behavior problems, the 
match on baseline achievement might not fully capture the inherent educational disadvantages of charter school 
students, thereby biasing our analysis against better performance from charter schools. 

We think that the fact that our study is situated in Milwaukee helps to reduce the threat of positive or negative 
unmeasured selection biases. As discussed above, many school choice options are available to parents even 
within the Milwaukee Public School (MPS) system. Highly motivated parents, or parents of students who are 
struggling in their neighborhood public school, can and likely do seek out alternative placements for their child 
within MPS. These options include vouchers to attend private schools, magnet schools, MPS charter schools, 
and open enrollment into other school districts. Since school choosers are present in both our charter and MPS 
matched comparison samples, concerns about self-selection bias when comparing student achievement gains 
across sectors are, to some extent, mitigated. 



Main Effects and Supplemental Analyses 

The primary goal of this report is to estimate the effect of independent charter school attendance on student 
achievement gains. However, a secondary goal of the report is to understand the specific mechanisms through 
which any effects might operate. Over the four years of this study, students in the MPS and independent 
charter samples likely experience a number of changes in family and school context, and some of these changes 
are likely to affect student achievement. An example we explore in the appendix is school switching. In this 
study, and many other studies, school switching has a negative effect on student achievement (e.g. Hanushek, 
Kain & Rivkin, 2004; Cowen et al, 2010). If there are differences in switching schools for charter school 
and MPS students, that may contribute to an explanation of the main effect which is the basic comparison 
on achievement growth between the two samples. Thus in what follows we present main effects based on all 
students in the samples, and then provide a set of supplemental analyses which refine and help us understand 
important variations in and possible explanations for our main effects. 
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Average Math and Reading Achievement 

We employ both descriptive statistics and multivariate methods to compare two- and three-year achievement 
gains of students in independent charter schools and comparable, matched students in Milwaukee Public 
Schools. Prior to any analysis, we first standardized the WKCE scale scores into z-scores using the MPS 
district means and standard deviations for math and reading.^ For all MPS students this procedure would 
produce an average z-score of 0 with a standard deviation of 1.0. Our samples deviate from these norms at 
baseline in that the average standardized student score is below 0, a fact which further confirms that the students 
in both the Independent Charter Sample and MPS Matched Sample are educationally disadvantaged relative to 
the average MPS student. These normalized z-scores are used throughout the analysis. After constructing these 
standardized scores, we first compared mean gains in standardized scores for independent charter and MPS 
students across grades and subjects. 

In Tables 3 and 4, we report one-, two- and three-year math and reading achievement gains for students in 
our charter and MPS samples. These results are broken out by grade level to examine the variation in student 
learning gains by sector across the different grades. The change calculations are created by using the student’s 
grade in 2009, where for example, students in grade 3 at baseline are in grade 6 in 2009. Using a student’s 
grade in 2009, we estimate the mean change score for charter and MPS students in that grade. Since 2009 is 
the outcome year, a one-year change is defined as subtracting the 2008 score from the 2009 score. Similarly, a 
two-year change is computed by taking the difference of the 2009 score and the 2007 score and a three-year 
change is the difference between the 2009 and baseline scores.® After finding the average change score for 
each sector, we take the difference between the two scores to determine whether gains in math favor charter or 
MPS students. 



7 We computed normalized z-scores by grade level in all four years for reading and math. For example, the formula for 
ZMath2007 in Grade 3 was ((Grade 3 ScaleMath2007- Grade 3 MPS district mean scale score)/(Grade 3 MPS district standard 
deviation)). 

8 The one-year change score is missing for 1 0th graders because they were in 9th grade in the baseline year and therefore no 
WKCE baseline score is available for them. 
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In general, the grade-specific results, which are presented in Tables 3 and 4, are a mix mostly of positive charter 
effects and some cases of no significant differences.’ One exception to this general trend is MPS students in 
grade 8 make greater gains in math than charter school students after one and two years. 

For charter students who reach grades 6, 7 and 8 it appears that these students after three years make 
considerably greater gains in math than similar MPS students. Students in grade 6 also make greater two-year 
gains in this subject. In reading, students in grades 6, 7, and 8 make greater gains than MPS students in reading 
after two and three years. There are no statistically significant differences in achievement among the 10* graders 
in the independent charter and MPS samples across the years in both math and reading. In addition to the 
grade-specific results. Table 3 illustrates that three-year achievement gains in math across all grades (last row) 
are substantially larger for students attending independent charter schools than they are for similar students 
attending MPS. A similar trend is seen in Table 4 for reading where independent charter students across the 
full sample make statistically significant greater achievement gains in all years, with large gains for years two 
and three. 



9 In addition to analyzing achievement growth through mean comparisons, we also analyzed the data using Somer's D, which 
calculates the difference in the probability that a given independent charter school student will demonstrate more or less gains 
than a matched MPS student. The results of this analysis were substantively similar to the mean comparison and are available 
from the authors upon request. 



Milwaukee Independent Charter Schools Study: Report on Two- and Three-Year Achievement Gains 



March 2011 


9 







Table 3. Mean Math Achievement by Grade in Outcome Year, 2006-07 to 2009-10 



Grade 

2009 


Group 


One-Year 

Change 

(08-09) 




Two-Year 

Change 

(07-09) 




Three-Year 

Change 

(06-09) 








Mean 


s.e. 


Mean 


s.e. 


Mean 


s.e. 






Growth 


(diff) 


Growth 


(diff) 


Growth 


(diff) 


6 


Charter 


.285 




.240 




.317 






MPS Matched 


.045 




-.019 




.102 






(Difference) 


(.240)*** 


(.059) 


(.259)*** 


(.066) 


(.215)*** 


(.065) 


7 


Charter 


.102 




.215 




.451 






MPS Matched 


.059 




.156 




.260 






(Difference) 


(.043) 


(.050) 


(.059) 


(.060) 


(.191)*** 


(.062) 


8 


Charter 


-.660 




-.047 




.091 






MPS Matched 


-.506 




.071 




-.092 






(Difference) 


(-.154)*** 


(.059) 


(-.118)* 


(.063) 


(.183)*** 


(.062) 


10 


Charter 






-.023 




-.018 






MPS Matched 






-.032 




-.073 






(Difference) 






(.009) 


(.101) 


(.055) 


(.092) 


All Grades 


Charter 


-.085 




.114 




.223 






MPS Matched 


-.115 




.056 




.072 






(Difference) 


(.030) 


(.036) 


(.058) 


(.035) 


(.151)*** 


(.035) 



•"p<0.01, "p<0.05, *p<0.10 



Note: Two sample t-tests were run to test the significance of differences in average gains between our MPS Matched sample and 
Charter sample. Mean growth scores for each sector are rounded to the third significant digit. Response weights were included in the 
estimation of differences in means. 
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Table 4. Mean Reading Achievement by Grade in Outcome Year, 2006-07 to 2009-10 



Grade 

2009 


Group 


One-Year 

Change 

(08-09) 




Two-Year 

Change 

(07-09) 




Three-Year 

Change 

(06-09) 








Mean 


s.e. 


Mean 


s.e. 


Mean 


s.e. 






Growth 


(diff) 


Growth 


(diff) 


Growth 


(diff) 


6 


Charter 


.222 




.229 




.242 






MPS Matched 


.061 




.057 




.105 






(Difference) 


(.161)*** 


(.054) 


(.172)** 


(.070) 


(.137)* 


(.071) 


7 


Charter 


.098 




.322 




.329 






MPS Matched 


.048 




.114 




.131 






(Difference) 


(.050) 


(.052) 


(.208)*** 


(.057) 


(.198)*** 


(.061) 


8 


Charter 


.029 




.109 




.195 






MPS Matched 


.041 




-.027 




-.038 






(Difference) 


(-.012) 


(.052) 


(.136)** 


(.055) 


(.233)*** 


(.062) 


10 


Charter 






.063 




.101 






MPS Matched 






.035 




-.006 






(Difference) 






(.028) 


(.089) 


(.107) 


(.086) 


All Grades 


Charter 


.112 




.187 




.221 






MPS Matched 


.049 




.052 




.058 






(Difference) 


(.063)** 


(.030) 


(.135)*** 


(.033) 


(.163)*** 


(.035) 



*"p<0.01, **p<0.05, *p<0.10 



Note: Two sample t-tests were run to test the significance of differences in average gains between our MPS Matched sample and 
Charter sample. Mean growth scores for each sector are rounded to the third significant digit. Response weights were included in the 
estimation of differences in means. 



Milwaukee Independent Charter Schools Study: Report on Two- and Three-Year Achievement Gains 



March 2011 


11 







Models for Math and Reading Achievement 

Using the dataset described above, we estimate the impact of independent charter school attendance on gains 
in math and reading controlling for student characteristics.^ Our analytic sample used to estimate the effect 
of independent charter school attendance on two-year achievement gains consists of 2,929 students for whom 
we have test scores at both baseline and two years after baseline. The analytic sample used to estimate three- 
year achievement gains consists of 2,274 students. To control for potential achievement differences by grade, 
we include grade indicator variables in all equations. We control for baseline achievement by including the 
student’s baseline (2006) test scores in both subjects. The basic models for estimating two- and three-year gains 
are represented by the following equations: 



(eq 1) 
(eq 2) 






+P3Y2006r,,+PA+p5X, + S 

+P3Y2006r.,+PA+p5X, + S, 



In these equations, for each student i, represents the effect of student enrollment in a charter school in 2006- 
07 (C=l) and P^and P^ estimate the impact of baseline math and reading achievement. With this specification, 
the contribution of the baseline test to the estimate of the second year test is unconstrained in that P^ and P^ 
can take any value.” P^ represents a vector of grade-specific contributions to the intercept and P^ represents the 
impact of a set of student- level characteristics, X ., such as gender and race/ethnicity. 

The outcomes of interest are 2008-09 and 2009-10 reading and math standardized scores on the WKCE for 
students in grades 5-8, and 10 in 2008 and grades 6-8, and 10 in 2009. Student characteristics included are 
those typically found in studies of charter school performance and they include free and reduced lunch status. 



10 We had 1,623 missing test scores for 2008 in math and 1,625 in reading. In 2009, we had 2,277 missing test scores in math and 
2,277 in reading. For students who switched sectors after taking tests in November 2007 and 2008, if we could locate their 
tests in 2008 and 2009 in the new sector, we included them in the analysis attributing their growth results to their initial sector 
placement. This is standard practice for "crossovers" in randomized field trials. Also in this case it is safe to conclude that most 
of the sector switching took place over the summer, thus the majority of learning occurred for this first year in their initial 
sector. In subsequent years, as crossover enrollment increases, we will handle analysis of crossovers in multiple ways. See Witte 
etc?/., 2010. 

1 1 Some researchers have used differences in test scores as the dependent variable by subtracting the baseline test score from 
the outcome year test score. However, if we want to model achievement growth controlling for prior achievement, this has 
the effect of constraining the effect of prior achievement to 1.0, which empirically is not the true parameter. Thus, we favor the 
estimation model in Equation 1. 
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exceptional education status (ExEd), race, and genderd^ English language learner (ELL) status was not included 
because there were very few students formally classified as ELL in the charter schools. The race indicator 
variable is coded as 1 for black and 0 for non-black, which serves as the reference group. We collapse racial 
groups other than blacks into the non-black category because there are substantially fewer whites, Hispanics, 
Asians, and Native Americans in the sample. To correct for potential asymmetric attrition from the charter 
and MPS samples, we use nonresponse weights that were constructed using observable student characteristics. 
However, the weighted results reported below are substantively similar to the results from estimating the same 
models on unweighted samples. The models account for the clustering of students within schools by employing 
robust and clustered standard errors. 

Results for Regression Models of Charter Impacts on Two- and Three-Year 
Math and Reading Achievement Gains 

The regression results in Tables 5 and 6 comparing two- and three-year gains of students in Milwaukee 
independent charter schools to those in MPS exhibit some findings of no difference but mostly findings of 
positive charter effects. We view Model 3, which controls for baseline test scores and student demographics, as 
our best analytic model and will focus our discussion on the results from that model. Table 5 provides two-year 
growth estimates. Being enrolled in an independent charter school is estimated to have a small but statistically 
significant positive effect of .063 standard deviations on two-year reading achievement gains when controlling 
for baseline test scores and student characteristics. Although positive, being enrolled in a charter school is not 
found to have any statistically significant impact on two-year math achievement gains. 



1 2 We acknowledge that participation in the federal free and reduced lunch (FRL) program can be an inconsistent measure of 
student poverty across grades and sectors, since older students are more likely to decline participation even if eligible and 
some schools outside of the traditional public school system choose not to participate in the program at all. Such problems are 
especially acute when comparisons are being made between the public and private school sectors (Peterson & Llaudet, 2006). 

In spite of these concerns, we use the FRL indicator in our models for three reasons. First, our comparison is between different 
types of public schools -- independent public charters compared to schools within the Milwaukee Public School system . 
Second, all of the schools represented in our sample do participate in the FRL program. Third, participation in the FRL program 
is the only proxy measure of student poverty available to us. Leaving the federal lunch program out of our models would have 
invited omitted variable bias of an unknown direction and magnitude to undermine our analysis. 

1 3 For 2008-09, there are 3,528 non missing blacks and 1 27 non missing non-blacks. In 2009-1 0, there are 3,21 8 non missing blacks 
and 1 14 non missing non-blacks. 
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Table 5. Two-Year Growth Models of Math and Reading Achievement for Average Charter Impacts, 2006-08 





Model 1- Charter Status 


Model 2- Controlling for 
Prior Test Scores Only 


Model 3- Student 
Characteristics Included 


Math 

2008 


Reading 

2008 


Math 

2008 


Reading 

2008 


Math 

2008 


Reading 

2008 


Charter 2006 


.056 

(. 143 ) 


.041 

(. 108 ) 


.067 

(. 058 ) 


. 071 * 

(. 037 ) 


.040 

(. 053 ) 


. 063 * 

(. 037 ) 


2006 Score - Reading 






. 187 *** 

(. 023 ) 


S 2 1 *** 
(. 029 ) 


. 170 *** 

(. 019 ) 


AIT' 

(. 027 ) 


2006 Score - Math 






. 566 *** 

(. 026 ) 


. 229 *** 

(. 021 ) 


. 540 *** 

(. 028 ) 


. 234 *** 

(. 020 ) 


Grade 5 










-. 454 *** 

(. 106 ) 


5 5 3 
(. 097 ) 


Grade 6 










-. 316 *** 

(. 106 ) 


- 459 *** 

(. 090 ) 


Grade 7 










. 205 * 

(. 111 ) 


-. 474 *** 

(. 090 ) 


Grade 8 










- 441 *** 

(. 111 ) 


- 509 *** 

(. 093 ) 


Grade 10 










-. 489 *** 

(. 112 ) 


527 - 

(. 093 ) 


Black 










-. 267 *** 

(. 076 ) 


-. 210 “* 

(. 062 ) 


Female 










-.027 

(. 026 ) 


113 

(. 025 ) 


ExEd 










(. 052 ) 


- 299 *** 

(. 047 ) 


Free Lunch 










-. 080 ** 

(. 033 ) 


-. 088 “ 

(. 036 ) 


Constant 


.014 

(. 040 ) 


-.037 

(. 042 ) 


. 109 *** 

(. 021 ) 


.026 

(. 020 ) 


. 739 *** 

(. 127 ) 


. 765 *** 

(. 117 ) 


N 


2929 


2927 


2929 


2927 


2929 


2927 


R' 


.002 


.003 


.493 


.540 


.566 


.560 


F 


6.61 


3.03 


508.8 


396.0 







p<0.01"*, p<0.05**, p<0.10* 



Tabled results control for race, gender, grade, free lunch, exceptional education In Model 3. Reference categories for these variables 
are non-black, male, grade 4 in 2008, no free lunch, and no exceptional education. Two prior test scores are also controlled for in all 
models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
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Table 6. Three-Year Growth Models of Math and Reading Achievement for Average Charter Impacts, 2006-2009 





Model 1- Charter Status 


Model 2- Controlling for 
Prior Test Scores Only 


Model 3- Student 
Characteristics Included 


Math 

2009 


Reading 

2009 


Math 

2009 


Reading 

2009 


Math 

2009 


Reading 

2009 


Charter 2006 


.132 

(. 123 ) 


.147 

(. 126 ) 


. 144 *** 

(. 040 ) 


. 156 *** 

(. 050 ) 


. 123 *** 

(. 033 ) 


(. 040 ) 


2006 Score - Reading 






. 231 *** 

(. 030 ) 


. 482 *** 

(. 030 ) 


. 191 *** 

(. 035 ) 


. 424 *** 

(. 027 ) 


2006 Score - Math 






. 495 *** 

(. 029 ) 


. 247 *** 

(. 023 ) 


. 501 *** 

(. 032 ) 


. 256 *** 

(. 022 ) 


Grade 6 










-. 321 *** 

(. 104 ) 


-.141 

(. 107 ) 


Grade 7 










-. 208 * 

(. 107 ) 


-.073 

(. 117 ) 


Grade 8 










-. 434 *** 

(. 104 ) 


-.190 

(. 126 ) 


Grade 10 










-. 502 *** 

(. 116 ) 


-. 221 ** 

(. 107 ) 


Black 










-. 206 ** 

(. 081 ) 


-. 377 *** 

(. 087 ) 


Female 










-. 110 *** 

(. 034 ) 


. 082 ** 

(. 032 ) 


ExEd 










-. 309 *** 

(. 073 ) 


-. 277 *** 

(. 060 ) 


Free Lunch 










-.026 

(. 032 ) 


-. 087 *** 

(. 033 ) 


Constant 


1 23 
(. 039 ) 


-. 078 * 

(. 041 ) 


.003 

(. 024 ) 


.029 

(. 026 ) 


. 675 *** 

(. 140 ) 


. 625 *** 

(. 145 ) 


N 


2274 


2275 


2274 


2274 


2274 


2274 


R^ 


.008 


.009 


.495 


.512 


.524 


.539 


F 


6.34 


13.4 


292.1 


204.0 







p<0.01*", p<0.05”, p<0.10* 



Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables 
are non-black, male, grade 5 In 2009, no free lunch, and no exceptional education . Two prior test scores are also controlled for In all 
models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
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Analysis of three-year achievement gains (Table 6) reveals evidence of a large positive effect of charter school 
attendance on both math and reading achievement. When controlling for baseline test scores and student 
characteristics, the effect of charter schooling on three-year math and reading achievement gains is estimated to 
be an increase of .123 and .112 standard deviations, respectively. The estimated positive effects of Milwaukee 
independent charter schools on student three-year achievement growth in both reading and math produced by 
our Models 2 and 3 are all statistically significant at the 99% confidence level. 

The control variables in the models perform as expected, giving us confidence in the reliability of the analysis. 
Consistent with prior research, students in both MPS and the independent charter sectors with higher baseline 
achievement (2006 test scores) have higher two- and three-year achievement gains than students with lower 
prior achievement. In contrast, black students in both sectors show lower two- and three- year achievement 
gains than non-black students in most models. Similarly, students who receive exceptional education services 
exhibit lower achievement gains compared to non-disabled students. Female students outperform males in 
reading in both years we analyze, but are found to achieve smaller three-year gains than males in math. Finally, 
low-income students perform less well than more advantaged students in most models. 

Figure 1 provides a graphical depiction of the effect of independent charter attendance on one-, two-, and three- 
year achievement gains in both math and reading. The plots represent the coefficient values in Model 3, and the 
bars the confidence intervals around those estimates. If the bars do not cross zero, we can reject the hypothesis 
that there is no difference between independent charter and MPS achievement growth for the relevant year. As 
is apparent, all the mean estimates favor independent charters, but only three years of growth in math and two 
and three years of growth in reading allow us to say with at least 90% certainty that the independent charter 
advantages are greater than zero. 
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Effect of Independent Charter Attendance on Student Achievement 

Point Estimate and 90% Confidence Intervai 



Math 



























Reading 



2007 



2008 



2009 



2007 



2008 



2009 



Year 



NOTE: Point estimates and confidence intervais are based on Modei 3 resuits in Tabies 5 and 6 
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As stated in the introduction, this report is divided into main effects and supplemental analyses that help explain and 
understand the primary results. In this section we address three factors that add to our understanding of the variance 
in the primary effects. They are the type of independent charter school, charter school effects on students at different 
achievement levels, and the effects for a subgroup of each sample - those who stayed in either charter schools or 
MPS schools for the full four years. A final analysis of students who switch school is included in Appendix B. 



Variation in Main Effects by Type of Charter Schoois 

In addition to understanding the main effect of attending an independent charter or a MPS school, we are also 
interested in the effects of two different types of charter schools. Four of the independent charter schools were 
initially private schools that changed school sectors hy converting to puhlic-school charters (i.e. conversion charters). 
The other 6 charters were either startup schools or former public schools (i.e. non-conversion charters).^'* There were 
only 395 students in our sample enrolled in conversion charter schools in 2008-09 and 304 students in 2009-10, 
compared to 1,029 students in the non-conversion charter schools in 2008-09 and 797 students in 2009-10.^® 

We capture and test for the differential effects of these two types of charter schools by estimating equations 3 and 4. 



(eq3) 
(eq 4) 



Y,oo8,rPo + P.CC,+P,NCC.+ P3Y,„„,^ 

Y,oo.rPo + PiCC,+P,NCC.+ P3Y3„„,^, 



+P4Y2006r.,+P5G,+ p6X, + S- 
+P4Y2006r,,+P5G,+ p6X, + £- 



In this specification we split the charter indicator variable in equations 1 and 2 into conversion charters (CC) and 
non-conversion charters (NCC),with the effects captured by estimating the P^and P^ parameters. The remaining 
variables are defined as in equations 1 and 2. 

There are more nuanced results when the main effects described above are further analyzed by the type of charter 
school. These results are depicted in Tables 7 and 8. Table 7 suggests that the small, positive effect of charter 
schooling on two-year reading achievement gains (Models 2 and 3 in Table 7) is driven primarily by conversion 
charter schools. The effect of conversion charter school enrollment on two-year reading achievement gains is 
positive and significant in all models. In contrast, attending a non-conversion charter school is not estimated to 
have any effect on two-year achievement gains in either math or reading. 



14 In most states, schools are called "conversion charters" If they are former traditional public schools that converted to charter school 
status. We use the term here to refer to former private schools in our sample because a substantial number of private schools (4) 
have converted to public charter status in Milwaukee. 

1 5 Tables with enrollment counts by grade for conversion and non-conversion charter schools are available upon request.. 
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Table 7. Two-Year Growth Models of Math and Reading Achievement for Conversion 
and Non-Conversion Charter Impacts, 2006-08 





Model 1- Conversion and Non- 
Conversion Charter Status 


Model 2- Controlling for 
Prior Test Scores Only 


Model 3- Student 
Characteristics Included 


Math 

2008 


Reading 

2008 


Math 

2008 


Reading 

2008 


Math 

2008 


Reading 

2008 


Conversion 
Charter 2006 


. 415 ** 

(. 207 ) 


371 *** 
(. 125 ) 


.141 

(. 089 ) 


141 *** 

(. 051 ) 


.103 

(. 082 ) 


. 118 *** 

(. 029 ) 


Non-Conversion 
Charter 2006 


-.081 

(. 133 ) 


-.085 

(. 099 ) 


.038 

(. 069 ) 


.044 

(. 044 ) 


.013 

(. 067 ) 


.040 

(. 050 ) 


2006 Score - 
Reading 






. 185 *** 

(. 022 ) 


. 520 *** 

(. 029 ) 


. 168 *** 

(. 019 ) 


. 471 *** 

(. 027 ) 


2006 Score - 
Math 






. 562 *** 

(. 027 ) 


. 225 *** 

(. 022 ) 


. 536 *** 

(. 029 ) 


731 

(. 020 ) 


Grade 5 










-. 456 *** 

(. 105 ) 


-. 555 *** 

(. 097 ) 


Grade 6 










-. 314 *** 

(. 107 ) 


-. 457 *** 

(. 088 ) 


Grade 7 










. 205 * 

(. 112 ) 


-. 475 *** 

(. 088 ) 


Grade 8 










-. 442 *** 

(. 111 ) 


-. 510 *** 

(. 093 ) 


Grade 10 










-. 481 *** 

(. 117 ) 


-. 520 *** 

(. 092 ) 


Black 










-. 227 *** 

(. 084 ) 


- 191 *** 

(. 057 ) 


Female 










-.029 

(. 026 ) 


. 111 *** 

(. 025 ) 


ExEd 










-. 262 *** 

(. 056 ) 


-. 306 *** 

(. 047 ) 


Free Lunch 










-. 066 * 

(. 036 ) 


-. 075 * 

(. 039 ) 


Constant 


.014 

(. 040 ) 


-.037 

(. 042 ) 


. 108 *** 

(. 021 ) 


.026 

(. 020 ) 


. 708 *** 

(. 141 ) 


. 739 *** 

(. 109 ) 


N 


2929 


2927 


2929 


2927 


2929 


2927 


R' 


.025 


.026 


.494 


.541 


.567 


.561 


F 


5.33 


5.08 


439.3 


331.6 







p<0.01***, p<0.05**, p<0.10* 

Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables 
are non-black, male, grade 4 In 2008, no free lunch, and no exceptional education . Two prior test scores are also controlled for In all 
models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
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Table 8. Three-Year Growth Models of Math and Reading Achievement for Conversion 
and Non-Conversion Charter Impacts, 2006-09 





Model 1- Conversion and Non- 
Conversion Charter Status 


Model 2- Controlling for 
Prior Test Scores Only 


Model 3- Student 
Characteristics Included 


Math 

2009 


Reading 

2009 


Math 

2009 


Reading 

2009 


Math 

2009 


Reading 

2009 


Conversion 
Charter 2006 


. 492 *** 

(. 147 ) 


. 550 *** 

(. 137 ) 


. 197 *** 

(. 073 ) 


. 258 ** 

(. 102 ) 


. 167 ** 

(. 064 ) 


. 178 *** 

(. 051 ) 


Non-Conversion 
Charter 2006 


-.011 

(. 108 ) 


-.014 

(. 115 ) 


. 123 *** 

(. 040 ) 


. 115 ** 

(. 049 ) 


. 105 *** 

(. 032 ) 


.084 

(. 053 ) 


2006 Score - 
Reading 






. 229 *** 

(. 030 ) 


. 478 *** 

(. 029 ) 


. 190 *** 

(. 036 ) 


. 422 *** 

(. 027 ) 


2006 Score - 
Math 






. 492 *** 

(. 030 ) 


. 241 *** 

(. 023 ) 


. 499 *** 

(. 032 ) 


(. 022 ) 


Grade 6 










-. 322 *** 

(. 105 ) 


-.143 

(. 106 ) 


Grade 7 










-.204 

(. 109 ) 


-.068 

(. 118 ) 


Grade 8 










-. 432 *** 

(. 105 ) 


-.187 

(. 125 ) 


Grade 10 










-. 500 *** 

(. 118 ) 


-. 220 ** 

(. 107 ) 


Black 










-. 189 ** 

(. 093 ) 


5 5 

(. 076 ) 


Female 










-. 110 *** 

(. 034 ) 


. 082 ** 

(. 032 ) 


ExEd 










-. 315 *** 

(. 073 ) 


-. 287 *** 

(. 061 ) 


Free Lunch 










-.015 

(. 033 ) 


-. 070 ** 

(. 035 ) 


Constant 


-.124 

(. 039 ) 


-. 079 * 

(. 041 ) 


.002 

(. 024 ) 


.027 

(. 026 ) 


. 649 *** 

(. 155 ) 


. 586 *** 

(. 141 ) 


N 


2274 


2275 


2274 


2274 


2274 


2274 


R' 


.035 


.044 


.496 


.515 


.524 


.540 


F 


6.49 


23.1 


237.4 


210.6 







p<0.01"', p<0.05”, p<0.10* 

Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables 
are non-black, male, grade 5 in 2009, no free lunch, and no exceptional education. Two prior test scores are also controlled for in al 
models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
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Table 8 provides further evidence of a positive effect of conversion charter attendance on achievement gains. 
Controlling for baseline test scores and student characteristics (Table 8, Model 3), conversion charter attendance 
is estimated to increase three-year achievement gains in math and reading by .167 and .178 standard deviations, 
respectively. Table 8 also indicates that, relative to MPS, attending a non-conversion charter school is estimated 
to increase three-year achievement gains in math by .105 standard deviations when controlling for baseline test 
scores and student characteristics. In reading, just controlling for prior test scores, students in non-conversion 
charters make .115 standard deviations greater gains in three-year reading achievement compared to MPS 
students. In summary, conversion charter schools lead the way in achievement gain differences, but non- 
conversion charter schools also outperform MPS schools over three years. 

Variation in Main Effects by Student Achievement Leveis 

A second supplemental analysis examines potential variation in charter school impacts using “quantile 
regressions.” We include these models because the charter impacts on student learning gains may not be 
constant among students with different levels of outcome achievement. Quantile regressions allow us to reliably 
estimate the effects of charter schools for students performing at different levels of achievement. We examine 
charter impacts at the 10*, 25*, 50*, 75* and 90* percentiles of the 2009 outcome achievement distribution. All 
of the controls included in Model 3 in Tables 5 to 8 are included in the quantile regressions. 

The impact of independent charter schools is generally strongest for students at the low end and middle part of 
the achievement distribution. Table 9 indicates that the effect of charter school attendance on two-year reading 
gains is twice as large for students at the 10* percentile of the distribution compared to students at other points 
of the distribution. Conversion charters are found to be especially effective at increasing reading achievement 
at the low end of the distribution. Although students in the middle parts of the distribution appear to benefit 
from charters in many cases, none of our estimates revealed statistically significant two-year effects of charter 
schooling on students at the upper end of the performance distribution (90% and higher). 



16 Results of the effect of conversion and non-conversion charter attendance on one-year achievement gains are not presented in 
this report, but are available in Witte eta!., 2010. 
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Table 9. Results of Quantile Regressions Estimating Charter Impacts on Two-Year Achievement Gains 





.10 


.25 


Median 


.75 


.90 


Ua\h (N=2929) 


Charter main effect 


, 026 (. 057 ) 


, 047 (. 028 )* 


, 050 (. 025 )** 


, 069 (. 023 )*** 


, 014 (. 033 ) 


Conversion charter 


. 137 (. 085 ) 


. 092 (. 043 )** 


. 079 (. 033 )** 


. 099 (. 037 )*** 


. 048 (. 052 ) 


Non-conversion charter 


-, 022 (. 059 ) 


, 033 (. 030 ) 


, 029 (. 024 ) 


, 062 (. 026 )** 


, 004 (. 038 ) 




Reading (N=2927) 


Charter main effect 


. 106 (. 051 )** 


. 048 (. 028 )* 


.051 (. 022 )** 


.051 (. 027 )* 


. 034 (. 034 ) 


Conversion charter 


. 214 (. 083 )‘* 


.071 (. 047 ) 


, 069 (. 033 )** 


, 066 (. 040 )* 


. 034 (. 051 ) 


Non-conversion charter 


. 094 (. 059 ) 


. 024 (. 034 ) 


. 047 (. 024 )** 


. 047 (. 029 ) 


. 036 (. 039 ) 



p<0.01"*, p<0.05**, p<0.10* 



Note: Test scores measured as standardized scores. Tabled results control for race, free lunch status, exceptional education, grade, 
and prior achievement. The point estimates for these controls are not included in the table but are available upon request. 



Table 10. Results of Quantile Regressions Estimating Charter Impacts on Three-Year Achievement Gains 





.10 


.25 


Median 


.75 


.90 


Math (N=2274) 


Charter main effect 


.271 (. 049 )*** 


, 178 (. 047 )*** 


. 097 (. 029 )**‘ 


, 083 (. 027 )*** 


, 105 (. 038 )*** 


Conversion charter 


. 347 (. 085 )*** 


. 239 (. 067 )*** 


. 096 (. 048 )** 


. 099 (. 045 )** 


. 067 (. 065 ) 


Non-conversion 

charter 


, 237 (. 057 )*** 


.141 (. 046 )*** 


, 099 (. 034 )*** 


, 084 (. 032 )*** 


. 116 (. 045 )** 




Reading (N=2274) 


Charter main effect 


.207{.062r' 


.081 (. 040 )** 


. 079 (. 028 )**‘ 


. 092 (. 031 )*** 


. 166 (. 035 )*** 


Conversion charter 


, 297 (. 095 )*** 


. 138 (. 061 )** 


.093 (. 045 )** 


, 077 (. 052 ) 


. 155 (. 051 )*** 


Non-conversion 

charter 


.101 (. 065 ) 


. 055 (. 041 ) 


. 064 (. 032 )** 


. 089 (. 037 )** 


. 178 (. 037 )*** 



p<0.01"', p<0.05**, p<0.10* 



Note: Test scores measured as standardized scores. Tabled results control for race, free lunch status, exceptional education, grade, 
and prior achievement. The point estimates for these controls are not included in the table but are available upon request. 



The charter school impacts for lower achieving students are very clear after three years (Table 10). This table 
illustrates that independent charter school attendance generally has its largest effects at the low end of the 
achievement distribution. Specifically, charter school enrollment is estimated to increase math and reading 
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achievement by between one-fifth and a Httle more than one-fourth of a standard deviation for students at the 
10* percentile of the achievement distribution. 

Conversion charters are again found to be particularly effective at increasing three-year achievement gains at 
the low end of the student performance distribution. In addition, and consistent with the results presented in 
Tables 7 and 8, conversion charters are generally found to be more effective at increasing achievement than 
non-conversion charters. This finding applies across most of the distribution of achievement, although for 
certain quantiles of students the difference in the effects of conversion and non-conversion charters on student 
achievement are not significantly different from each other. 

Variation in Main Effects for Students Who Stay in the Same Sector for Four Years 

The main effects for this study used all students in charter schools and an equal number of matched MPS 
students based on their initial sector attendance in 2006. However, students switch schools and sectors, and 
some are lost to the study. These are ongoing problems for longitudinal observation and random assignment 
studies. A key issue is the attribution of treatment effects. For example, if a student is initially in a charter 
school, but then spends the next three years in an MPS school, how do you attribute their learning gains? There 
is no single answer. We can stick with the initial condition, as we have in the main effects study and as is done 
in randomized field trials, or we could drop “crossovers,” or we could weight the relative exposure to charters 
or MPS. 

Another method is to estimate the results only for those students who remain in the same sector for the 
duration of the study. Following this approach, we estimated results for a subgroup of “stayers”. Stayers are 
students who remain in the same sector, independent charter or MPS, for the three or four years of the study. 
These estimates are a sensitivity analysis to control for attrition and those who crossover between sectors. 

The results of this analysis are presented in Tables 11 and 12. The results illustrate that, among those students 
who remained in their initial sector from 2006-07 until 2008-09, charter enrollment had a positive effect on 
two-year reading achievement gains on the order of .143 standard deviations when controlling for baseline test 
scores and student characteristics (Model 3, Table 11). This effect is more than twice the size of the main effect 
(.063) for the full sample (Table 5, Model 3). There was also a positive charter effect of .168 standard deviations 
on two-year math achievement gains just controlling for prior test score. There was no statistically significant 
difference between the independent charter and MPS samples in two-year mathematics achievement gains 
controlling for student demographics (Table 11). That was also true for the main effects analysis. 
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Table 11 . Non-Sector Switching (Stayer) Two-Year Growth Models of Math and Reading Achievement for 

Independent Charter Schools on Average, 2006-08 





Model 1- Charter Status 


Model 2- Controlling for 
Prior Test Scores Only 


Model 3- Student 
Characteristics included 


Math 

2008 


Reading 

2008 


Math 

2008 


Reading 

2008 


Math 

2008 


Reading 

2008 


Charter 2006 


.211 

(. 174 ) 


.159 

(. 124 ) 


. 168 ** 

(. 072 ) 


. 145 *** 

(. 046 ) 


.118 

(. 074 ) 


. 143 *** 

(. 045 ) 


2006 Score - 
Reading 






. 197 *** 

(. 026 ) 


. 529 *** 

(. 030 ) 


.163 

(. 023 ) 


. 469 *** 

(. 027 ) 


2006 Score - 
Math 






. 586 *** 

(. 023 ) 


. 227 *** 

(. 020 ) 


. 561 *** 

(. 024 ) 


(. 020 ) 


Grade 5 










-. 626 *** 

(. 138 ) 


-. 736 *** 

(. 099 ) 


Grade 6 










-. 459 *** 

(. 132 ) 


-. 617 *** 

(. 102 ) 


Grade 7 










.062 

(. 135 ) 


-. 660 *** 

(. 100 ) 


Grade 8 










-. 577 *** 

(. 131 ) 


-. 684 *** 

(. 095 ) 


Grade 10 










5 5 3 
(. 135 ) 


-. 626 *** 

(. 110 ) 


Black 










-. 220 *** 

(. 087 ) 


(. 068 ) 


Female 










.011 

(. 030 ) 


1 25 *** 

(. 032 ) 


ExEd 










-. 295 *** 

(. 054 ) 


-. 330 *** 

(. 053 ) 


Free Lunch 










-.060 

(. 042 ) 


-.040 

(. 042 ) 


Constant 


.018 

(. 041 ) 


-.039 

(. 043 ) 


. 108 *** 

(. 022 ) 


.017 

(. 021 ) 


. 816 *** 

(. 158 ) 


. 883 *** 

(. 123 ) 


N 


2416 


2416 


2416 


2416 


2416 


2416 


R' 


.011 


.010 


.534 


.549 


.606 


.574 


F 


2.84 


2.84 


402.0 


266.2 







p<0.01"', p<0.05”, p<0.10* 



Tabled results control for race, gender, grade, free lunch, exceptional education In Model 3. Reference categories for these variables 
are non-black, male, grade 4 in 2008, no free iunch, and no exceptional education. Two prior test scores are also controlled for in all 
models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
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Table 12. Non-Sector Switching (Stayer) Three-Year Growth Models of Math and Reading Achievement for 

Independent Charter Schools on Average, 2006-09 





Model 1- Charter 
Status 


Model 1- Controlling for 
Prior Test Scores Only 


Model 2- Student 
Characteristics 
Included 


Math 

2009 


Reading 

2009 


Math 

2009 


Reading 

2009 


Math 

2009 


Reading 

2009 


Charter 2006 


. 384 *** 

(. 121 ) 


.362 

(. 127 ) 


. 300 *** 

(. 045 ) 


. 278 *** 

(. 064 ) 


. 251 *** 

(. 038 ) 


. 231 *** 

(. 055 ) 


2006 Score - 
Reading 






. 206 *** 

(. 031 ) 


. 483 *** 

(. 033 ) 


. 141 *** 

(. 031 ) 


. 403 *** 

(. 030 ) 


2006 Score - 
Math 






. 506 *** 

(. 034 ) 


(. 026 ) 


. 520 *** 

(. 035 ) 


. 251 *** 

(. 027 ) 


Grade 6 










-. 394 *** 

(. 131 ) 


-. 282 *** 

(. 099 ) 


Grade 7 










-. 224 * 

(. 135 ) 


-. 183 * 

(. 099 ) 


Grade 8 










-. 485 *** 

(. 133 ) 


-. 317 *** 

(. 100 ) 


Grade 10 










-. 458 *** 

(. 144 ) 


-. 218 * 

(. 111 ) 


Black 










-. 217 ** 

(. 102 ) 


-. 382 *** 

(. 081 ) 


Female 










-. 098 ** 

(. 038 ) 


. 099 ** 

(. 038 ) 


ExEd 










-. 425 *** 

(. 061 ) 


-. 366 *** 

(. 060 ) 


Free Lunch 










.025 

(. 044 ) 


-. 064 * 

(. 038 ) 


Constant 


-. 107 ** 

(. 041 ) 


-.045 

(. 044 ) 


.000 

(. 025 ) 


.041 

(. 028 ) 


. 682 *** 

(. 167 ) 


. 725 *** 

(. 122 ) 


N 


1697 


1697 


1697 


1697 


1697 


1697 


R' 


.042 


.039 


.509 


.533 


.545 


.568 


F 


5.58 


6.84 


185.4 


144.7 







p<0.01"*, p<0.05", p<0.10* 



Tabled results control for race, gender, grade, free lunch, exceptional education in Model 3. Reference categories for these variables 
are non-black, male, grade 5 In 2009, no free lunch, and no exceptional education. Two prior test scores are also controlled for In all 
models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
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Examining three-year achievement gains among students who remained in the same sector from 2006 
until 2009, Table 12 reveals that charter attendance led to statistically significant gains in both reading and 
mathematics achievement. This positive effect is present across all model specifications.^^ As above, the effects 
for stayers are approximately twice the size of the effects for the full sample. Thus it seems very clear that the 
positive charter school effects reported in the main section of this paper are much larger when we only analyze 
students who have been in their initial sectors for three or four years. 



CAVEATS 



There are three primary issues that affect the explanatory power of these analyses. First, the results in this report 
only encompass the first four years of a five-year evaluation of Milwaukee’s independent charter schools. It is 
possible that the results will change in future reports and require different substantive conclusions to be drawn. 

Second, as in many studies of schools with low income students, there are missing data due to sample attrition. 
Issues of sample attrition are explored in substantial detail in Appendix A, but we note here that 19.7 percent 
of panelists were missing in 2008, with 14.0 percent ofMPS sample members missing and 25.4 percent of 
independent charter sample members missing. In 2009, the attrition rate for the full sample increased to 26.8 
percent, with 18.7 ofMPS sample members and 35.0 percent of independent charter sample members unable to 
be located. Although these numbers are lower than expected, and lower than in a number of other studies, they 
could raise concerns that the attrition was non-random. In theory, this could affect accurate overall population 
estimates of gains, but because there were few baseline test differences between missing students from either 
sample, we believe our sample comparisons are accurate and should remain so (See Appendix A). Nevertheless, 
we weighted the analytic sample for these differences accordingly. Thus we feel confident that attrition has been 
handled well and that we will have the capacity to continue to produce reliable estimates of independent charter 
school effects in Milwaukee in future years even if the non-random nature of that attrition becomes worse. 



1 7 Descriptive statistics for variables used in the multivariate analyses are available from the authors upon request. 
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This report is the second of three reports about the performance of independent charter students in Milwaukee 
compared to a matched sample of students in the Milwaukee Public Schools. The results of our analysis of two- 
and three-year achievement gains generally support the existence of an independent charter schooling advantage 
in student achievement growth in math and reading in Milwaukee. 

The detailed conclusions of this report are: 

• Based on three years of student achievement growth, charter school students outperformed MPS 
students in both reading and mathematics after controlling for baseline achievement and other student 
characteristics. These results were statistically significant with more than 99 percent confidence. 

• Drawing upon our earlier study of charter school gains (Witte et ah, 2010), we see a clear pattern of 
positive charter school effects growing over time. There was little consistent evidence of differences 
in achievement gains between charter and MPS students after one year. The second year growth was 
better for charters in some models and for some tests, but not for others. After three years, a sizable 
independent charter school advantage was apparent in all of our analyses. 

• Supplementary analysis revealed that conversion independent charter schools - schools that converted 
from private schools - as a subgroup did better than MPS schools in all three growth years. In the third 
growth year non-conversion charter schools also did better than MPS schools in mathematics but not 
necessarily in reading. 

• Further analyses indicate that after three years charter schools appear to have the greatest positive impact 
on students at the lower end of the achievement distribution. 

• When looking specifically at “stayer students”, or those who remained in their initial sector (charter or 
traditional) for four years, the performance of charter stayers was much higher than that of MPS stayers 
after three years. 

As indicated in the Executive Summary and throughout the analysis, we caution that the results in this report 
are based only on three years of estimated achievement gains. Subsequent reports have the potential to alter the 
general findings and conclusions. 
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Appendix A- Study Attrition 

Table A-1 . Baseline Student Characteristics for Non Missing and Missing Students in 2008-09 and 2009-10 





2008-09 


2009-10 




Non Missing 


Missing Students 


Non Missing 


Missing Students 


Average Mean 
Baseline Math 


-.135 


-.107 


-. 151 ** 


-.070 


Average Mean 
Baseline Reading 


-.089 


-.065 


-.095 


-.053 


Female 


1,816 


441 


1,642 


615 


(%) 


( 49 . 7 ) 


( 49 . 1 ) 


( 49 . 3 ) 


( 50 . 3 ) 


Black 


3 , 528 *** 


847 


3 , 218 *** 


1,157 


(%) 


( 96 . 5 ) 


( 94 . 2 ) 


( 96 . 6 ) 


( 94 . 7 ) 


White 


87 ** 


34 


78 ** 


43 


(%) 


( 2 . 38 ) 


( 3 . 78 ) 


( 2 . 34 ) 


( 3 . 52 ) 


Hispanic 


32 ** 


15 


18 * 


29 


(%) 


( 0 . 88 ) 


( 1 . 67 ) 


( 1 . 47 ) 


( 0 . 87 ) 


Native American 


4 


3 


3 * 


4 


(%) 


( 0 . 11 ) 


( 0 . 33 ) 


( 0 . 09 ) 


( 0 . 33 ) 


Asian 


4 


0 


0 


4 


(%) 


( 0 . 11 ) 


( 0 . 00 ) 


( 0 . 00 ) 


( 0 . 12 ) 


Free Lunch 


2 , 853 *** 


589 


2 , 609 *** 


833 


(%) 


( 79 . 6 ) 


( 65 . 5 ) 


( 78 . 3 ) 


( 68 . 2 ) 


ExEd 


460 


97 


433 *** 


124 


(%) 


( 12 . 6 ) 


( 10 . 8 ) 


( 13 . 0 ) 


( 10 . 2 ) 


Baseline Grade 3 


716 ** 


145 


676 *** 


185 


(%) 


( 19 . 6 ) 


( 16 . 1 ) 


( 20 . 3 ) 


( 15 . 1 ) 


Baseline Grade 4 


711 *** 


126 


658 *** 


179 


(%) 


( 19 . 5 ) 


( 14 . 0 ) 


( 19 . 8 ) 


( 14 . 7 ) 


Baseline Grade 5 


733 *** 


124 


703 *** 


154 


(%) 


( 20 . 1 ) 


( 13 . 8 ) 


( 21 . 1 ) 


( 12 . 6 ) 


Baseline Grade 6 


631 *** 


83 


444 *** 


270 


(%) 


( 17 . 3 ) 


( 9 . 23 ) 


( 13 . 3 ) 


( 22 . 1 ) 


Baseline Grade 7 


450 *** 


237 


476 ** 


211 


(%) 


( 12 . 3 ) 


( 26 . 4 ) 


( 14 . 3 ) 


( 17 . 3 ) 


Baseline Grade 8 


414 *“ 


184 


375 *** 


223 


(%) 


( 11 . 3 ) 


( 20 . 5 ) 


( 11 . 3 ) 


( 18 . 3 ) 


TOTAL (N) 


3,655 


899 


3,332 


1,222 



p<0.01”*, p<0.05", p<0.10* 

Note: The average mean baseline math and reading scores are normalized z scores. Stars indicate that non missing students are 
different from missing students. 
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Missing and Non-Missing Students 

The overall percent of missing students from the study during 2008-09 is 19.7 percent. Missing students in the two samples are not the same as 
non-missing students. The relevant data are portrayed in Table A-1 . Generally, missing students in 2008-09 exhibit baseline achievement which is 
not statistically different from non-missing students; but they are more likely to be white and Hispanic. Non-missing students are more likely to 
be black and on free lunch. Although these differences are not large, we corrected for these characteristics in our statistical models presented above. 

The overall percent of missing students from the study during 2009-10 is 26.8 percent, with 18.7 of MRS sample members and 35.0 percent of 
independent charter sample members unable to be located (Table A-2). Again, given that we anticipated sample attrition at 20 percent annually, 
these attrition statistics are actually somewhat better than expected. 

Missing students in 2009-10 score higher on average in math and the same in reading compared to non-missing students; and these students are 
more likely to be white. Non-missing students are more likely to be black, on free lunch or have a disability. Full results are presented in Table A-1 . 



Missing Students Between Sectors 

As indicated in Table A-2, there is considerable difference between the two samples, with 14.0 percent missing in MRS and 25.4 percent missing 
from the independent charters. Students in both independent charters and MRS may have left for private schools in the MRCR program or may 
have moved out of the city of Milwaukee. We have tracked students between sectors (i.e. "crossovers") using test score and enrollment data, but 
we undoubtedly missed some students who will be recovered in subsequent years. We do not have the data to track students into private schools, 
which are likely to account for more missing in the charter schools. Given that we anticipated sample attrition at 20 percent annually, we are 
optimistic about these results. 

Students who are missing in the two sectors for 2008-09 differ on some student characteristics. Compared to MRS students, independent charter 
students are less likely to be black, on free lunch and have a disability; and more likely to be Hispanic (Table A-2). In addition, these students are 
more likely to score higher in math on average than their MRS counterparts, but there are no differences in reading. 

Similar to the attrition pattern after two years, missing charter schools students after three years are less likely than comparable MRS 
students to be black, on free lunch and have a disability. In addition, these students are more likely to score higher in math on average than 
their MRS counterparts. Again, differences are small for the proportion of black students and all differences are adjusted with controls in our 
explanatory models. 
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Table A-2. MPS vs. Independent Charter Attrition Statistics for 2008-09 and 2009-10 





Missing in 2008-09 


Missing in 2009-10 




MPS Matched 


Independent 

Charter 


MPS Matched 


Independent 

Charter 


Average Mean 
Baseline Math 


-. 208 ” 


-.051 


-. 201 *” 


.000 


Average Mean 
Baseline Reading 


-.121 


-.034 


-.078 


-.040 


Female 


152 


289 


212 


403 


(%) 


( 47 . 7 ) 


( 49 . 8 ) 


( 49 . 8 ) 


( 50 . 6 ) 


Black 


309 ” 


538 


41 5 *** 


742 


(%) 


( 96 . 9 ) 


( 92 . 8 ) 


( 97 . 4 ) 


( 93 . 2 ) 


White 


8 


26 


9 * 


34 


(%) 


( 2 . 51 ) 


( 4 . 48 ) 


( 2 . 11 ) 


( 4 . 27 ) 


Hispanic 


r 


14 


^ *** 


17 


(%) 


( 0 . 31 ) 


( 2 . 41 ) 


( 0 . 23 ) 


( 2 . 14 ) 


Native American 


1 


2 


1 


3 


(%) 


( 0 . 31 ) 


( 0 . 34 ) 


( 0 . 23 ) 


( 0 . 38 ) 


Asian 


0 


0 


0 


0 


(%) 


( 0 . 00 ) 


( 0 . 00 ) 


( 0 . 00 ) 


( 0 . 00 ) 


Free Lunch 


231 *** 


358 


310 ” 


523 


(%) 


( 72 . 4 ) 


( 61 . 7 ) 


( 72 . 8 ) 


( 65 . 7 ) 


ExEd 


50 *” 


47 


66 *” 


58 


(%) 


( 15 . 7 ) 


( 8 . 10 ) 


( 15 . 5 ) 


( 7 . 29 ) 


Baseline Grade 3 


68 ”. 


77 


86 *” 


99 


(%) 


( 21 . 3 ) 


( 13 . 3 ) 


( 20 . 2 ) 


( 12 . 4 ) 


Baseline Grade 4 


56 ” 


70 


70 


109 


(%) 


( 17 . 6 ) 


( 12 . 1 ) 


( 16 . 4 ) 


( 13 . 7 ) 


Baseline Grade 5 


51 


73 


69 *” 


85 


(%) 


( 16 . 0 ) 


( 12 . 6 ) 


( 16 . 2 ) 


( 10 . 7 ) 


Baseline Grade 6 


35 


48 


66 


204 


(%) 


( 11 . 0 ) 


( 8 . 28 ) 


( 15 . 5 ) 


( 25 . 6 ) 


Baseline Grade 7 


r /•*** 

56 


178 


68 


142 


(%) 


( 17 . 6 ) 


( 30 . 8 ) 


( 16 . 0 ) 


( 17 . 9 ) 


Baseline Grade 8 


53 ” 


131 


67 * 


156 


(%) 


( 16 . 6 ) 


( 22 . 6 ) 


( 15 . 7 ) 


( 19 . 6 ) 


TOTAL (N) 


319 


580 


426 


796 


(%) 


( 14 . 0 ) 


( 25 . 4 ) 


( 18 . 7 ) 


( 35 . 0 ) 



***p<0.01, **p<0.05, *p<0.10 

Note: The average mean baseline math and reading scores are normalized z scores. Stars indicate MPS Matched different from 
Independent Charter statistics based on a two-tailed t-test. Percentages are rounded to the third significant digit. 
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Appendix B- School Switching 

The explanatory results for models including school switching are described below. We view school switching as a downstream mediator meaning that 
switching can be considered as a part of the treatment (independent charter school attendance). When we include school switching, the charter effect 
becomes non-significant in most of our main explanatory models. The one exception is the conversion charter effect after two years (.066 standard 
deviations) which remains positive and significant. For considerations of space, these nonsignificant results are not tabled, but are available from the 
authors upon request. Where there are positive effects of independent charter schools on average and by type, they may be partially explained by the 
greater stability of charter attendance after two and three years. Specifically, there are fewer charter school students switching schools after two and 
three years (See Table B-1). 

Results from regressions of"stayers", students who remain in the same sector over the four years of the study, show some significant advantages 
for charter students after controlling for school switching. Specifically, charter students on average and conversion charter students make greater 
two- year gains than MRS students in reading. Likewise, students in conversion and non-conversion charters make higher three-year gains than MRS 
students in both subjects. See Table B-2 for full results. 

B-1. Non-Sector Switching (Stayer) Estimates of Two- and Three- Years Gains for Charter Students 

Controlling for Switching from Initial School 





MPS Matched Switchers 

(%) 


Independent Charter Switchers 

(%) 


2006-08 


1,365 


672“* 




(69.7) 


(39.6) 


2006-09 


1,586 


794*** 




(85.7) 


(53.6) 



Stars indicate MPS Matched statistics are different from Independent Charter statistics at p<0.01'", p<0.05", p<0.10*. Switched schooi 
in this case means students switched a schooi either between or within sector after two years (2006-08) and after three years (2006-09). 

B-2. Descriptive Tables of Baseline Students who Switch Schools in the MPS and Charter Sectors 

after Two- and Three- Years 



School Type 


Estimate including School 
Switchingfor 2008-09 

(N=2416) 


Estimate including School 
Switching for 2009-10 

(N=1697) 


Math 


Charter 


.089(.082) 


.156(.047)*“ 


Conversion Charter 


.144(.094) 


.146(.060)“ 


Non-Conversion Charter 


.063(.105) 


.161 (.054)*“ 


Reading 


Charter 


.099(.047)“ 


.149(.063)“ 


Conversion Charter 


.120(.038)*“ 


.154(.051)*“ 


Non-Conversion Charter 


.089(.063) 


.146(.084)* 



p<0.01***, p<0.05**, p<0.10* Tabled results control for race, gender, grade, free lunch, exceptional education and switching 
schools. Reference categories for these variables are non-black, male, grade 4 in 2008 for two-year gains model, grade is 5 in 2009 for 
three- year gains model, no free lunch, no exceptional education and not switching schools, respectively. Two prior test scores are also 
controlled for in all models. Robust standard errors are estimated to account for the clustering of students within particular schools. 
Estimates for student characteristics, prior test scores and grade dummies are available upon request. 
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